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ABSTRACT 

\^ A comparative case study of seven universities 
initially focused on: (1) whether, and hon program evaluation systems 
differ from^ other systems approaches, such as management infarmatian 
and proqtam budgeting; (2) desirable degree of overlap betweeu 
•valuative recommendations and administra tive • action ; and (3) 
limitations of program evaluation. Site visits raised several . issua 
statements. Pirst, what was the effect of staff ing-- specif icdily, the 
use ot outside consultants, . and the methdld;*^)^^ selecting evaluators? 
Second, did institutional tradition, camj^us' governance, and state 
coordination alter the evaluation? Third,,' were the effects of formal 
• valuation stroctures, (checlcllsts, |)rjbtocols, schedules, task 
fiiprc«s) a healthy boost to either institutions or to subsequent 
evaluations? Pourth , was there an advantage to linking program 
•valuation systems to budget systems? An assessment of thes* issues 
led to the conclusion that formal systems in higher education were 
new and significant modifications of more traditional program rjaview 
activities fluch as accreditation, national ratings, ins tit utx6nal 
research, and institutional self -study. (Author/CP) 
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Preface 



Systematic evaluation efforts developed especially for campus level 
program evaluation have become a conmon feature of many universities. In gen- 
eral, these evaluation systems have sought to operational ire some notion of 
accountabilny. consumer protection, or responsible planning. Evaluation had 
been a standard feature of campuses for several years^ these newer structures 
were merely more visible systems of providing information for and fac.il ty par- 
ticipation in administrative decision making. This new thrUU also sought to 
combine evaluation with existing budgetary, personnel and /Uagerlal systems. 

The current discuss ion is, grounded i;i a survey of several previously 
established review activities found in higher education! These more traditional 
review activities included institutional and program accreditation, national 
ratings, institutional research, operations research, institutional traditions, 
and institutional self-study. These traditional activities have been contrasted 
with systematic efforts at several university campuses across the United States. 
Campuses included in the study are: 

^ University of Washington - Seattle 

University of Oregon - Eugene " 
University of Cal ifornia - Berkeley 
University of Michigan 
University of Minnesota 

State University of New York at Albany. Buffalo 
University of Illinois - Urbana ^ 

Site visits to these .campuses led to the development of several issue statements. 



^^he current paper is organized around four of these issues, including 



"Printed in U.S.A." 



cfLJ^^r ° evaluation systems as important a con: 

sideration as n appeared? Did it matter if the evaluation 
team was xompjetely in-house? Was the level of involvement 
in Itifl? ^"V^"^'u^ populations the crucial deteminant 
in staffing? Were there important implicatipns for evalu- 
ation models where staffing was prescribed? 

The elements of institutional traditions or mystiques, cam"- 

^"^^'""^1 external . and 
thf^nf ? combined to influence evaluation systems, 

the role of evaluation, and associated sanctioning processes 
Did these elements alter the' resul tant evaluation? 

itll f^^i^^ °^ elements Of formal evaluation structures 
(checklists. (Protocols, schedules, task forces . etc. ) . a 
healthy boost either to institutions or subsequent evalu- 

Were there identifiable advantages to joining formal evalu- 

^° a planning 

system? Did this structure have implications for the stat- 
ure of operational or short term planning? 
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Issues IN HIGHER EDUCATION ^>ROGRAMJVALU/^r ION ' 
Ijitroduction * . , •» . • ' , 

Assessment Of the desirability, feasibility, and effectiverxess of ' 
campus ev|luation systems depends in part upon identifying whether these 
.attempts at progra'm, review weVe a real and significant new evaluation force op 
merely a rearrangement of old activities, This review supports the notion 
: that formal systems of program evaluation in higher education were new and 

significant modifications of more traditional program .review activities. 
: This is not to imply that program reviews conducted in this formal 

♦node were npt in large part made up of procedures used previously in tradi- 
tional Evaluation activities. One need only examine two or three evaluation • 
^ystems to discover that the differences in systemic reviews were not the result 
df ^dramatically new techniques -or novel approaches. These formal systems do, 
however, -.reflect several . departures from traditional evaluation efforts. 

' , Among the more obvious departures, systematic efforts were broader 
based. Cut across disciplinary* and departmental borders, i/ivol ved mul tiple 
rather than single' or dual sanctions, fallowed contemporary notions of popular 
values, and recast the roles of many central campus figures. The writings 
•wMch marked the establishment of these systems helped to makexlear the al- 
tered percep^ion<,,o^ evaluations : These perceptions were demonstrated, by 
statements oV what the "system" might accomplish. ^ 

Goals of the N^w Systems ' ^ 

_At th^ University of Michigan program review was expected to pro- 
vide a broader base of information than was previously available to adminis- 
trators in making decisions. The Pr6gram Evaluation Committee Report of . 
October 19-73 indicated that: , 
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-AJ^, present, budgetary constraints, from both state and 
.federal sources forte deans, department heads, directors 
and executive* officers , to make hard decisions regarding 
, present and future operations, of the University ' These 
hard choices are made today based upon a variety of infor- 
mation and decision processes. One objective of the Pro- 
gram Evaluatio;i Committee is to identify a set Of proce- 
dures which may be of assistance to the key decision-makers, 
as they foeus upon an issue fof'al location of limited fiscal 
resour(?es. (p. 4) ' ' ■ 

^ • A further aspiration 'pf the University of Michigan^ system of 
evaluation'was imbedded in the following statement. "The question -is not 
wljether the University should conduct evaluative activities; rather, the " 
question: becomes whether the University should undertake evaluations the way 
th^t they are presently conducted or in a more efficacious manner." (Univer- 
sity of Michigan, p. 4) 

Other goals for systematic review efforts were enumerated by the 
University of Minnesota. These included: ' 

1. To insure that each program conducts a periodic self-assessment 
of its status. V . ' 

' ■ ■ ' :.. ' ■■ ^ 

^ 2. To allow each department thie opportunity to present in terms of 
data and descriptive material its own perspective on its role in the University, 

3. To provide the deans of the Graduate School andjhe Institute of 
Technology a data base and supplementary in/grmatfon to aid them in fnternal 
planning, in preparation of budget requests ,> and in meeting the requirements 
of accountability. (Brodbeck;^.1974j ^ 
In addition, the systematic reviews of the Univers'fty of Minnesota attempted 
to consolidate the Unlversitx^ approach to academic , planning ; to facilitate 
and sharpen periodic reviews; and to assist the faculty in examinirtg and eval- 
uating their program fields aided by>iVers.ity caTleagues in related fields and 
by external visitors from other i^nstitutions and pertinent enterprises ( Grad- 
uate Progra m Review Statement , revised SepiSnber', 1973) 



Many of these (|oals of Yoniul jbrocjram roview could have been ac- 
'complished b'y.less formal evaluative activities. .The first goal stated by 
the University of Minnesota ' perhaps best summarized the perceived advantage 
of the newer "fonnal'.' structures over "traditional evaluation." Ratified 
formal pi^ocedures provided an increased measure of assurante that evaluative 
activities would be carried but regularly and in 4 prescribed"nianner. 

I * 

Mr 

^ . Many/program evaluation systems, made use 0/ admini'strfitors , pro- 
.gram faculty, students, faculty from outside the program area, and often con- 
sultants external to' the university. With th'is broad participation more -per- 
spectives were expressed! although discussion^ were more diffuse. 

Relation of New Systems to Other "Systems" Approaches 

These systems of review contrasted sharply with other "systems" ap- 
proaches applied to higher education,, such as rtianagement information systems 

and program budgeting systems. I/1 program' review systems, communication pat- 

y ■ ■ 

terns were complex_ap<t.' intertwined. There was a sense in which these patterns 

recreated the patterns of communication typically found in the university's 
normal routiqe. Other "systems" approaches sought to alter and streamline 
these communications- patterns making them regularized, coded and efficient. 
Program review allowed respondents to step out of traditional coimuni cation 
pottcri, with tha expressed intent oi' describing the status quo, not replacing 
it. Repl2c"e'-,ent Cc?me later^os a judg'^icnt activity. ' 

A capstone review c-ctivity used at the UniversiV of Michigan per- 
haps best illustrates this point. At the University of Michigan the Vice- 
President for Academic Af fairs^rticipated in a capstone briefing near the 
end of program review activities. Tnis briefing also involved the college - 
dean, department chairpersons, and staff from the of f ice -of budgets rfnd pla/^- 
ning. -This briefing was an opportunity for the Vice-President for Academic 
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Affairs to become familiar with college needs and future. directions. 

• 3 sense, this b^•iefing was different from normal communication 
• . patterns because the chain of command called for discussions between depart- 
njent heads and deans, between d^ans and the vice-president, but not for 
group discussions between these parties. : T^iis variation did not seek to re- 
•place either the established communication patterns or the chain of comnand 
.however; it merely accentuated these patterns, reduced the slippage and con- 
centrated discussion on self-study results 

' • • r 

■ Limitations of the New Systems * 
\ Numerogs limitations of ^he systems apfirpach to program evaluation 

have also become evident. Progr6mi rev,iew systems tended to emphasize par- 
.... V ' 

ti€ipat.ron of faculty, students, admit^istratops , aY\6 conmunity groups. As a 
result committees or task groups were'the <iominant format: The comnittee ap- 
pro4ch often did not pinpoint admihrstratjve deficiencies. Cotmvittee mem- 
bers,_ through negotiation and compromise tended to generalize administrative, 
deficiencies to other aspects of program functioning. Many assumed a less 
personal-ized^ystems approach which concentrated on data inputs and outputs, 
.would have provided more definitive and prescriptive recomnendations to over- 
come administrative deficiencies. 

. ySecond-ly, program review systems, again because of their dependence 
on conmittee -work, reflected uneven effort within the range of topics covered 
• in the review. The dynamics of comnittee functioning accounted for some of this 
unevenness, but it is difficult to separate this from the fact that all com- 
mittee members do not function with the same level of proficiency. 

A third major limitation of program review 'systems was that major 
effects of- program review activities were not observable or, even when observed, 
reportable. Many individual faculty members had only meager information about 

ERIC V 7 • ; 
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the efforts of the other faculty in thei*' departments. Lch of the effect of 
program review has been the result of the familiarization of faculty with 
program activities of which thej^ had formally hff(i litt/e knowledge. In the 
University of .Michigan example'Ja major result of program review came from. 
■ the department head beS'ng involved in the discussion between the dean and 
Vice-President for Academic 'Affairs or the Vice-President for. Academic Affairs 
observing interpersonal dynamics, between deans and department heads, 'such 
small deviations frdm normal operating procedures are difficult* to spot anf 
to measure or describe, but may significantly alter the unit^ evaluated. ' 

A fourth limitation observed nationally has been the failure of 
these evaluation systems to, meet the expectatioris held, by participants. The 
wide scope of these Systems made such varied demands on description and ana- 
lysis activities that it was impossible to cover all the values held by par- 
ticipants. Consensus did not permit all view points to be appropriately por- 
trayed. thus several points of view were not reflected in final evaluation 
reports. 

A fifth limitftion was illustrated by the failure of evaluation 
conmittee. attempts to prbpose solutions incompatible with normal funding 
patterns. Because evaluation systems were based on the development of inter- 
. disciplinary solutions, there was considerable tension between global images ' 
developed by coniiiittees,>\^d specific recommendations yvhich were drafted. 

Many questioned whether such systems were .control led by faculty, 
administrators', or both. Bennis (1976) and others illustrated, that the - 
.forces acting on higher education te/ided to fragment the authority of admin- 
Istrators, leaving thern less effective and more vulnerable. Administrators 
sought to Obtain support; without further erosion of their control of univer*- 
slty activities. 'As a-result, administrators tended to 'use program "review*' 
systenfs as a way to broaden their base of input and win Support for adminis- 
^j tratlve decisions. V " ^' 

ERIC . 8 
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The wider involvement which canie with program review systems caused** 
increased public scrutiny of administrative decisions. Tn adSition the strug-\ 
gle for control in higher education led to deman/s that evaluation recommenda- 
tions be reviewed by both faculty and administrators before adoption or widespread 
circulatjeR,. Th0 broad involvement, which characterized these review sys- 
tems,' created multiple roles for individuals involved in program review. Fac- 
ulty members became linked to the "administration, coordinatijig board or what- 
ever agency sanctioned the review ibrocess/ Thus as program evaluation, systems 
rewa'rded compl«lance with campus values, "the faciilty. student or connunity mem- 
bers involved in evaluation became messengers of those values. An institution 
clear in its mission and goals was certain'to have exemplary models as partic- 
ipants in the program review process, and throughVecoHpendations ai, analysis 
of the committee campus values were perpetuated". ; 



T 



Finally, systems of prograJ^ eval uation aggr.avated/th^ overlap of 



administration and faculty because of* the involvement thes^' systems fostered. 
Having spent considerai)le time and" energy in coitriittee woyk drafting evaluative 
^fotnnents and reconmendations, faculty typicaVly beqame d^irouS that evaluation 
material be used, not misused. One outcome of this desilre was a .struggle for 

' ' ' * / 

control over implementation of reconmendations. Ego inVolvement often dictated 

/ -v 

that a faculty committee move into the pol itical/admirvl'strative arena to sup- 
port, defend, and clarify its comments',' documentation, and reconmendations, 
even though the committee had Icnown al^ along that its status Was strictly toj 
kbe advisory. 



Issue Statement No. 1 
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Was the staffing of evaluation systems an important consid- 
eration? Did it matter if the evaluation team was completely 

' 'nnnM?^tfLc"5L^^^ ^f^r^^I inyoTvement of faculty and student • 
populations the crucial determinant in staffing? 
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in!Eortjin«_oiLitaf^ The staffing of evaluation systems defined 
. patterns of conmuni cation, and participation, human resources available, and 
perspectives in_volved in.the reviews. It -is not clear whether this influence 
was/exclusively the result of the selection criteria which guided the staffing 
of program review committees, or one of several other factors which influenced 
* prog-ram evaluations. Not the least of these 'factors was the standard adminis- ^ 
trative structures which typical ly organized central administrations. Because 
the chief academic off icer was also usually the chief budget officer, the lo- 
cation of evaluation staff with4n the university h-ierarcKy was cri ticaV. In 
addition the perceived distance of evaluation from the chief academic officer 
determined the credibility and perceived autonoriy. of^the review system. Be- 
cause evaluatioo typically begins wit+i academic prograhis. mpre specifically 
the expensive, graduate degree programs, additional constraints are imposed 
on the program review activity by oragnizational dynamics. 

Bl^^d' Si tuationalVactors. other stable influences mediated the 
^ staffing question! 'Dress^l (1976) has illustr^ted^the inappropriateness 

of .using standing cormiittees in institutional self-study or program, review. • * . 
Standing cormiittees tended to bs inhibited and'ego involve^! with maintaining 
. the status quo. While t-his inhibition did not make rational review impossi- 
. ble. it nonetheless substantially reduced coimiittee effectiveness. Standing 
.committees spent valuah^ time covering old ground and too often relied on 
old.channels for soliciting . infmjmation. These colthittee were often unsuitable for 
the tasks of 4nstitutional self-study Jand program review. Standing cormiit- 
TeeTbecause of prior involvements had a jump on the 'identification of, issue? 
and information sources; but this was often negated by reduced objectivity, reduced 
creativity and the birfding effect of stpong fnternar interpersonal' relation- 
$H1ps and roles. 

erIc ■ • . 
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. Executive and policy conriittees also wo^^shown to be. inappropriate 
fo. self-study activities. should be noted; however, that often the ap. 
pointment of a new coordinating c6nr,ittee for institutionaJ review activities 
posed a threat to .the established cormiittees and their traditional or stat- 
uatory responsibilities. It was often the ca^e that systematic review activ- 
ities all but eliminated the role of these conmittees. In several cases a ^ 
special central pffice or committee was used to coordinate all evaluative ' ' 
activities associated with periodic or systematic prdgrdm review. „ / ' . 
^ This central coordinating tonnittee was typically an internal com- 

mittee, -coordinating design, impleraentation. and an^ilysis of eval ufft^ion ac- • 
.tivities. Such activities included. the effp^ts of 4commi-ttees* taskVorces. 
• administrators and institutional staff .' • Bel ovy. this ^ross level . mqjjel s .for 
systematic review varied with program.constitJents an<^ other forpes outside-^ 
the program or institution. , The pattern of involv^ement was detemined by the 
type of review involved, whether it is primarily' forf budget r^allocatibn. 
»long ran^e planning, or accreditation. However, . there were seveVal , general ' 
parameters which shaped the staffing issue. ! " ^ 

Use of Oytside Consultants. A focus of considerable attention was ' 
on the value of outside consultants in the program review process. The issues " 
regulating the use of external- consultants were two: first, the evaluative 
question^ to be asked and. second, the available resources. Models ranged 
from a modified accreditation approach which"used external consultants and 
site visits, to a wholly internal approach which used no regular involvement 
of outside consultants. - ' , 

A further and overlapping consideration was the audience of the eval- 
uation and Hs requirements. The audience dictated which "evaluative perspec^ 
tlve was most relevant. State coordination was n^e 'likely to involve com-^ 

ERIC \* H 



11 

/ ■ 

parison with other institutions both in st^te and out-of-state becduso 
it'requlred assurances ^batp-ublic needs be efficiently' met. Exclusively 
internal reviews had to grapple with questions of internal allocations and 
. ' hence had to be concerned with how central a program was to the university's 
scope andjnission regardless of Jts s.tat^e or national reputation. 

. ,^ Cost was''\he secortd major fac.tor affecting the use of outside con- 
sultants, for example, the State u\i vers ity of New York at Al bany used external 
panels extensively for review of graduate programs. Consultants Were priroar- 
ilydrawn fr6m the. eastern seaboard ani typically spent one to two days on* 
campus. Each of these panels of .three scho\ars external to the university, 
nfpre^a^d an ^average university investment of $1500; ' In an insti tution com- 
mi.tted to reviewing 10 to 20 percent of Its programs annualiy.Vexternal con- 
sultan^ may cost from 10 to 40 thousand dollars annually, 
^ \\ ' ^Often institutions attempted to combine" internal and external re- 

viewers. Various models of this sort existed. = These ranged from appointing 
single external consultants to work with internal conmittees-. to choosing ^in- 
dependent Internal (STid external comittees cooTdinated by a third intern^li 
coninittee. ■ ^ . . ' 

Selerctlon Strategies. Perhaps no other element of formal evaluation 
structures more accurately reflected the prevailing style^f an institutiori , 
^han the ipethod of selection of evaluation participants. Models of selection 
•ranged from constituency elections to the appointment of- "faculty st^rs" by. ' 
chief academic oVficers. Equally reflective of institutional style was the 
^question of involvement and selection of students and .cornnunity participants. 
Pre<J|BdentS existed of near-student dominated Coninittees-for,exampl.e, The ' 
Study of -Graduate Educati on at Stanford -a^ well as of systems involving no 
(, fOipial student, in put. The involvement of faculty, students, administrators. » 

.' ^ ... 4 ' 
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and others in program review has tended to follow the established institutiqnal 
pattern of involvement , for these groups. - . . 

An txample from the University of Illinois . The system of program ^ 
review at the University of Illinois - Urbana was named' the Council on Program 
Evaluation. (COPE) The operations of the COPE Council showed clearly several 
effects of staffing patterns'-jen program evaluation systems. With regards to 
staffing, COPE was a tvyo-tiered system. The coordinating tier included an ap- 
pointed faculty conriittee and administrative staff support. This coonittee 
was appointed by the Vice-Chancellor for Academic Affairs who was the chief 
academic officer of the campus. Appointments to the counci 1. were rrfade with 
•the concurrance of the governance system^ and included both faculty and Stu- 
dent members; ^ . , 

The second tief of the evaluative system operated at the department 
or unit level and worked as either a departmental self-study comnittee or a 
COPE appointed task grO»p composed of . uni versi ty faculty, and students from both 
Inside and outside the department. No exlei;nal consultants were routinely used 
in program review. 

Thus the COPE system was characterized by the automatic involvement 
of the Vice-Chancellor of Academic Affairs, an Independently Si?lected coordi- 
nating conmlttee, student representation, single rather than joint reviews, 
and Infrequent use of external consultants. 

These parameters of staffing Illustrated and defined a system of 
•power relationships. The Vice-Chancellor having been a driving force In the 
establishment and operation of COPE, maintained appointment powers as COPE v^, 
implemented. These powers were mediated only by the concurrance which he was 
to s«ek from governance. The establishment of the chief academic and budget." 
officer as a central figure in the selection process Increased the planfulness 



of COP£ selection and operation.- . ' - 

4 

Planfulness and faculty involvement in' Krge -part, may be credited ' 
with contril)u,ting a broad plaining and e ducat ioii'al base to the COPE activity. 
Faculty participation fostered by COPE i ncreased "^campus real ization of , , 
the far-reaching effects of evaluation .and managemerrt decisions. The- ul timate 
goal of r^lacing year-to-year expediency wi th cohesive analysts, planning, 
^and management was" forwarded by this' staff ing arrangement. 

A domino effect was also observable in COPE'S staffing pattern. An 
obvious side effect of the central role of the Vice-chancellor of Academic 
Affairs in establishing and maintaining COPE, was Se undiluted influence which 
he had over the features of t^re' review structure. As stated earlier, these 
influences had to co-exist with institutional traditions and the prevailing 
political climate. 

Another factor in the determination of the pattern of\artici pation 
i^as the .interlocking effect of COPE affiliation. COPE membership required 
status as a respected member of the campus community. Senior campus citizens 
whose high professional standing was commonly accepted were nominated to COPE. 
The cohesiveness bf. campus perspectives were reflected in these nomt ' 
inations. At Urbana, the campus "sees itself as the pre-eminent publ icly-suo- 
ported graduate-level irwtitution within the stat4 . . . (and further) sees 
Itself as distinct from other state-supported institutions because of Its 'em- 
phasis on research and seminal scholarship." 

The disciplinary and rank cross-pollination which did exist sup- 
ported an appearance 'of balance essential for the operation of a central campus 
evaluation activity. As the most^vious element of the evafuijtiorK process 
COPE was the most likely focus of attacks on evaluative recofimendations. COPE 

4 

h«<f to »ppe<r to be balanced wUh respect to dlsctpHnes and perspectives 
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found wUhin the institution. . 

While a cohesive committee perspective on preferred academic stvle 
facilitated evaluative discussions and multiplied the reliability of evalu- 
ative judgments, a point of some concern was that other perspectives became 
• disenfranchised, it was important .to maintain a threshold of agreement orf 
campus missions, a threshold v<hich was neither ^narrow, or totalitarian. The 
common -perception on the Urbana campus was that COPE was not unduly restric- 
tive. But some have speculated that COPE reflected a rigid fragmentation of 
academic pi^rsuits on the campus. COPE has been accused of supporting- an im- 
plicit junior partner relationship for many dep^irtments , based' on the estab- 
lished scholarly values of the campus. The crucial qu^tion, of how much 
cohesion was necessary for productive ar^re1i£|ble evaluations was never com^ 
pletely answered although it was clear tWt campuses which had never clarified 
their mission were less receptive sites for systematic program review. 

* Two characteristics of COPE mediated^ the effects of cohesive campus 
values. COP£. occupied a fairly nebulus zone between faculty and administration, 
-^s a result it was unclear what status should be ascribed to COPE. Secondly, 
the very nature of institutional self-study usually prevented COPE from un- 
covering unknown weaknesses In^the campus armor. As a result COl'E tended to 
Influence the tenor -rather than the substance of campus debates. 

Faculty participation In program review Increased participation of 
the faculty In university affairs and stimulated the sharing of Information. 
The primary success of faculty participation was the broadening of Investigation, 
dialogue, and human resources. Such participation did considerably l^y towards 
enfranchising faculty on major Institutional decisions. Many In fact argued 
faculty Involvement should never erode administrative prerogative. And In ' 
fact. COPE did not result In a totally participatory system of decision making. 
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COPE represegted faculty participation in^formation gathering, probl, 
definition, and even alternatives identification and analysis, but not in 
allocation, implementation, or follow-up. 

Because of its peculiar stance COPE was also vulnerable to CO- V 
option and thus to reduction of the benefit's of broad participation. In COPE 
co-option resulted from the unquestioning hannony which linked COPE to the 
administration which was to implement or ignore ifs evaluative recormienda- 
tions. Co-options reflected in the degree to which the evaluative group 
speculated, not on the wisdom but marlcetabil ity of its recormiendations . The 
conmittee did not want to spend hours deliberating recormiendations which 
would be buried in stacks of reports housed in attic storage. Thus, the 
temptation was great to anticipate the leanings of campus\administrators. 
COPE tried to reduce this possibility by recruiting strong campus figiires 
who collectively had stature independent of recormiendation imple-hientation. 
. Yet. -because committee stature was enhanced by the serious attention given to 
Its recommendations,- the danger of co-option persisted. / 

Co-option posed another threat to the quality of institu- 
tional self-evaluation since the redistribution of knowledge and general stim- 
ulation which should accompany self-study was easily choked. The expanded 
consciousness of departmental faculty expected from se'lf-study activities 
needed the support and definition of public statements. If the coordinating 
committee became too closely aligned with administrators It rendered Itself 
incapable of giving this support. The staffing patterns of fonnal program 
^view -systems therefore had much to do with whether the subsequent review 
was geared for hartow administrative audiences. "and whether reviews encouraged 
fragmented perspectives, repeated traditional campus values or promoted aca- 
demic growth. ^ 



It*' 



Issue No. 2 • :/ 

The elements of institutional traditions or mystiques, 
campus, governance structures, (both internal and external), 
ands^ate coordination combined to influence eValuation sys- 
teirfsTthe role V evaluation, and , associated sanctioning pro- 
gesses. Should these elemer^ts have altered the evaluations? \ " 

* Traditions, mystiques .and oJ:her features of the >^gher education 

milieu encouraged campus cohesion. As discussed earlier, ^s cohesion was 

a necessary though not sufficient condition for viable program evaluation. 

Specifically campus traditions of student involvement, facility participation 

and administrative prerogative. were central staffing parameters. 

Weaknesses of^Formal Evaluation Pr^rams . One' area Ti^j^hich campus 
cohesion weakened formal evaluation activities was in the establishment of the 
evaluative criteria to be applied in the. reviews. Ir establishing the criteria 
by which programs were to^be assessed, it was important that long standing 
campus mores be put into perspective. Restriction of the range of criteria 
and the subtitutton of campus traditions for evaluation criteria together im- 
posed rigid controls on program reviews. Accepting quality research as a pos-' 
ttive criterion, and further restricting quality research to scholarly academic 
research clearly reduced evaluative activities to very narrow activitifi^ based 
on a predesigned system of valuing? where only traditional esoteric research 
was to be highly regarded. • 

In evaluation, and particularly in institutional self-study, it was 
very ttisy to alldw, ease of measurement, strength of tradition, or influence 
'of governing boards to.determine not only what constitued evidience but also 
how that evidence was to be valued. This was true even though popular notions 
of accountability assuwd very cohesive institutional goals which could be 
/"behaviorably" defined. . 

This perspective on the relationship between campus cohesion and 



evaludnion supports several observations rega^ii^ng those formal evaluation 
structures which have tended to promote tra'ditional perspectives. The^first 
such observation is that such systems tended> to be inteVnally oriented/ Givpn 
. the two primary evaluation questions" of internal and external comparison, Wa-i 
dUional structures were mor^ likely^to emphasize internal questions. Ratio- ' 
nales for controversial recommendations were steeped in the rhetoric of tra- 
di.tional campus values "and surrounded by an air of matter-of-factness . ' Of.tnen 
this independence was manifested in attempts to make the evaluation activities 

d regular cyclical canrpus dcti^vit^. -This is not to lay that all cycli: ' 

cal evaluations were tradition dominated, but merely that tradition dominated 

sy^>tGms sought the stability and security which regularization brought. 

' Beyond the campus cohesion question, several additional elements of 
c^impus expression connected to institutional self-image and mystiques or tra- 
ditions are noteworthy. As the' flagsh,ip campus of the University of Illinois, ^he 
campus had been considered by many to. be the hub of state supported higher od- 
ucdtion. Traditions of scholarly research, suppo'rted an image of campus 
integrity: One outcome was the progressive stance of campus administrators 
which found expression in program evaluation, lirbana administrators, operating 
on a do-i t-to~yoursel f-beforr?-it 's-done-to-yeu phil6sophy, began program. eval - 
uatioo without particular external pressure for program review. In estai)lish- 
ing COPC, these administrators were abh? to further reassert the integrity of 
the campus community. By beirtg among the /irst in the state to engage in sys- 
tematic self-eva.luation Urbana assured its independence; COPL would shape future 
state initiatives in program evaluation' ra^h(?r than being shaped' by them.. This 
has in fact occurred. The lilinois Board of iJigher Education ' s 'state wide pro- 
yram evaluation effort has required no significant changes in COPF.. 

Urbana 's second Independent motivation for program review' was the 
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campus ddmirfistration's incremental i St perspective on ch/nge. Following 
■ campus precedents, COPE was cVeated as a tool for reallocation, and was not 
meshed with either the budget cycl^ or other budget mechanisms.. Its impact 
was intended to.be steady and reasoned.- The subtle press^fAf pe'er review, 
"wars to be its majo'r power. *COPE was intended to provide the Vice-Chancel 1 or 
for Academic Affairs the information he required for administrating the campus, 
but pcr'hdps mbre important there was hope that through the processes of faculty 
input enough ,informatton could be shared within departments to make adminis- 
, trdt.iv^ action leSi frequently required. One central campus administrator 
complained that administrator's were forced to give 90 ^cent of th'eir^'time to 
what amounted to 10 percent of their r*esponsibiTi ty. Systems like COPE had 
the advantage of studying their subject without critical urgency. The over- 
all goal was to facilitate incremental tflovement towarcV; change and avoidance 
of the impass, and^ntrophy found in organization unprepared to deal with un- 
fprscen circumstances. 

I';f>uo Statement No. 3 



Were the effects of..-ElQments of formal eval uation Jstuc- 
tures. (checklists, protocols, schedules, taCV forces, etc.), 
a healthy boost either to institutions or to subsequent eval- 
uation?' 



\ 



I do not share the opinion that formal ^valuation activities in 
higher education were cosmetic changes designed to insure commitierit to pro- 
Tram review. • • . - 

Although formal evaHuative structures used fairly trad ftional instru- 
ments and strategies they added a longitudinal dimensfon. Typica 1 ly .program 
reviews wore repeated every rive to ton years, within every part of the uni- 
verTity. Ihis certaint/ has caused institutions to look for ways of using 
the vast amounts of -comparative data available. Programs have been asked to 
idcTitlfy peers with whom they might be compared, and program recormiendatlons 
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made f-ive to ten years earlier became starting points for a second ro|nd of • 
evaluations.. I 

\ ' * ' 

AnticipatiQn of evaluation carried addi tional momentum and influence 

when it included Ijnowledge of Xhe format to be used. The organization of 're- 
view ^formats provided a modeffor standing cormi ttees , and filing systems. 
However, many expressed concern^ that program functions would become too ' 
standardized. Many felt too much emphasis was on how to get the program 

into the compaftihients defined by/-{h^ stliAdard format. 

, For campuses vyith a fairly cohesive notion of what they are about, 
■these protocols and questionnaires further 'sol idified an already pervasive 
self-image. Programs were bombarded with subtle pressures to confomr to the 
st^andard- image implied in the format's, and to the cri terion .they reflected. , 
Sometimes altering the delicate balance of perspectives caused shifts in pro- 
grams as they attempted to become aligned with the perceived central ;nandate. 

At Urbana, for^ ex-ample, the establ ishment of COPE ushered into ex^s- «y 
tence a new survey of student majors (Program Evaluation Survey-PES), a task 
-force listing of categories of evaluation, a 14 category departmental self- 
study survey, standardized faculty activity sunnaries, seven standard tables 
for:,.depicting' instructional costs figures and faculty effort statistics, a 
department chairperson evaluation system, a standard unit resource' outl ine , 
and numerous special data. anaVysis formats. Each of these elements carried 
a coded image of an ideal department. Faulty activity sunnaries contained 
seven major headings, including such measures of national visability as ed- 
Itorial duties, reg1ona.l/,,at1onal offices held, major honor and professional ' 
society memberships, scholarly output,'a recent researeh support sun,«ary, years 
of experience, publications, and a university service comriittee summary. 

These protocols operaUlonalized dominant campus values and.rein- 
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forced these values because, once^released. they tended to be only selectivel 
revised in response to pointed crftieis.s. Thus over several years depaVt- " 
. "«nts and department faculty came.to expect that thejeseven categories of 
. . "tivity would be impocfjnt in COPE deliberations. 

COPE has-been criticized for allowing its standardized programatic 
-inputs to.result in a depersonalization of the review process. Department 
chairpersons have Judged COPE less responsive than .ight be desired. Dissat- 
isfaction was^higher in departm-ents which viewed themselves as not fitting 
the_ standard campus mode. ■ Attempts have been made to ali*r this image ap- ■ 
parently with so«e success Judg''ing by c«s from campus governance, indi- 
cating confidence in the ability of COPE council members to weigh carefully 
, the implications/limitations of such standardized infonnation requests. In 
addition some attempt was made to supplement fonnal responses wfth direct 
departmental information gathered by both COPE staff and council members. ^ 

Although the criticism of inflexibility has been addressed, two 
other questions remain unanswered. First, is the question of whether stan- 
dar^zed fomatf allow units to portray the'a'ctual flow of their transactions, 
or stimulate the generation of bureauVatic dialogue. On this issue two 
dominant perspectives have been articulated, both in support of standarized 
formats. Oneposition was premised on the notion that units had a sto^'to 
.tell, whiV. questionnaires and pro'tocols simply illicif. Program statements 
^e„ were^ot unduly influenced by Questionnaires. A second opinion contended 
.^thot quite t^e contrary occurred. In this perspective |,rograms wire viewed 
« having Interacted with protocols or questionnaires to develop a statement. 
The implication was that although instrumentation may affect the statements 
111 Iclted," standardization allowed comparison'. : ' . 
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jmga£ t o f . Sy _steffis . Systornatic rTyiei^s havei^^^covered' new problems 



for campus consideration. What such systems ha*e -tended .^d do U twofold 
First, the s-ystematJc collection of data has boths4.1.lDwed clearer statement 
of submerged campus issues and su^|,orted enlightened statements reg^ding 
the pervasiveness of these issues' ei-thar within a single unit or through 
several departments. Seconc^^^ getting these problems areas into the 
^PVblic arena^program evaluation has facilitated administrative action. In 
some cases this stimulation has' forced administrations Jo deal wit^* smoldering 
issues. V 

VhUe program evaluation stimulated analysis and dialogue in one 

■• \ " 

case, It ha^ limited communication in otfaers. \ Limitation occurred when stan- 

. " \ - % 

dard protocols reduced the need for contact bet;ween evaluative units and the 

departments being evaluated. Insftitutional ized^ormats which came to be 
view?cl as refined instruments for data^soTicitatiW tended to replace the 
hands-on stimulation of some traditional evaiuation\acti vi ties . They also 
reduced conversations to explanation of the ,various\orms and procedures in- 
volved. 

delated to the reduction of personal communication is a problem of 
insufficient communication, which was also linked lo standardized data for- 
mats. In an attempt to broader the applicability of protocbis, the specific 
Items used tended to bo less focused than might be expected.! Waco useful and 
unuseful information was received laced 'together. Standardized questionnaires 
worked as lightening rods and attracted •soroe.^troubl ing , some unuseful, and some 
jeopardizing information in their attempt to be complete. Distilling these re- 
ports resulted in a time lag for circulation of recommendeitions'. 

Another element of evaluation Influenced .by formal structures was 
confidentiality.. Collection of standi^rd data elements on every unit creatf^d. 



a ready nwrket for unofficial release of prograifi data which lead' to public 
comparisons based on narrow slices of evaluative data. In the case' of COPE, 
however, news leaks^ad been less a problem in the more standardized self- 
study' format than in the earlier task force system, perhaps because the stan- 
dardized self-study data was more routinely releasable. 

<^:1|inanswered question was the ability of these newer evaluation, , 
systems to recjf^^ewhat was generally known about- program quality with what was 
learned ab^ut the philosophy underpinning departmental activities. It was 
not clear from available evidence if cbmmon formats were capable of gener- 
ating the information required fqr making thi(_^nthesis. It was unclear 
how much of the synthesis was the result of the combined insight of COPE 
members and was hence dependent on quality COPE 'membership. 

J 

Issue Statement No . 4 • ^ 

^ ; 

Was there an identifiable advantage to joining formal 
evaluation systems^ to budget systems in order to form a plan- 
ning system? Did 'this structure alter the stature of opera- 
tional or short term planning? 

A primary bagef it of linking program evaluation systems with plan- 
ning or budgeting systems was that the evaluative act^ivity was thereby re- 
inforced. The power to reward7. departments with a higher priority for new 
monies or increased physical space added much stature to program review ac- 
tlvltles, and created an advantage for syste)B^^ltic evaluation activities 
over earlier evaluation efforts- which had been forced to operate- on the negative 
sanctions associated with peer pres5i!_re. The linking of evaluative and other func- 
tlona.1 systems Imparted positive rewards to, balance these- otherwise negative 
sanctions. This linking also Increased faculty involvement in important 
decisions, though this still largely depended on the direction provided by 
ev4T1fation participants. \ 
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In addition, the linking of evaluation with other systematic -effbrts 
. set dominant campys perspectives in an even' stronger position. The imposition 
of powerful perspectives was hard to avoid when the criteria were used in. pro- 
gram, review, long rang^ planning, and budget decisions. Integretate\ systems 
of this type had great potential for tightening- up campus planning and had to 
be closely monitored because of their repressive potential. 

In summary, linking evaluation to other activities has provided the 
potential for preventative maintanence. With systematic evaluation problem 
areas h^ve been addressed incrementally by various segments of the university 
before they had the opportunity to develop into university-wide concerns. The 
primary difficulty has been that program evaluation got further involved in . 
academic management. On campuses which moved too far in this direction, eval- 
uation systems developed which attempted to replace the statutorily created'! 
mechanisms for university management. 
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